{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(use/api)=\n",
    "\n",
    "# Python API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This page outlines how to utilise the cache programatically.\n",
    "We step throught the three aspects illustrated in the diagram below:\n",
    "[cacheing](use/api/cache), [staging](use/api/project) and [executing](use/api/execute).\n",
    "\n",
    "```{figure} images/execution_process.svg\n",
    ":width: 500 px\n",
    "\n",
    "Illustration of the execution process.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "The full Jupyter notebook for this page can accessed here; {nb-download}`api.ipynb`.\n",
    "Try it for yourself!\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialisation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "import nbformat as nbf\n",
    "from jupyter_cache import get_cache\n",
    "from jupyter_cache.base import CacheBundleIn\n",
    "from jupyter_cache.executors import load_executor, list_executors\n",
    "from jupyter_cache.utils import (\n",
    "    tabulate_cache_records, \n",
    "    tabulate_project_records\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we setup a cache and ensure that it is cleared.\n",
    "\n",
    "```{important}\n",
    "Clearing a cache wipes its entire content, including any settings (such as cache limit).\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "JupyterCacheBase('/Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/.jupyter_cache')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache = get_cache(\".jupyter_cache\")\n",
    "cache.clear_cache()\n",
    "cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n",
      "[]\n"
     ]
    }
   ],
   "source": [
    "print(cache.list_cache_records())\n",
    "print(cache.list_project_records())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(use/api/cache)=\n",
    "\n",
    "## Cacheing Notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To directly cache a notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbCacheRecord(pk=1)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "record = cache.cache_notebook_file(\n",
    "    path=Path(\"example_nbs\", \"basic.ipynb\")\n",
    ")\n",
    "record"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will add a physical copy of the notebook to tha cache (stripped of any text cells) and return the record that has been added to the cache database.\n",
    "\n",
    "```{important}\n",
    "The returned record is static, as in it will not update if the database is updated.\n",
    "```\n",
    "\n",
    "The record stores metadata for the notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'description': '',\n",
       " 'hashkey': '94c17138f782c75df59e989fffa64e3a',\n",
       " 'created': datetime.datetime(2022, 1, 12, 15, 15, 27, 255299),\n",
       " 'accessed': datetime.datetime(2022, 1, 12, 15, 15, 27, 255312),\n",
       " 'data': {},\n",
       " 'uri': 'example_nbs/basic.ipynb',\n",
       " 'pk': 1}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "record.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{important}\n",
    "The URI that the notebook is read from is stored, but does not have an impact on later comparison of notebooks. They are only compared by their internal content.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can retrive cache records by their Primary Key (pk): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[NbCacheRecord(pk=1)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.list_cache_records()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbCacheRecord(pk=1)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.get_cache_record(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To load the entire notebook that is related to a pk:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "CacheBundleOut(nb=Notebook(cells=1), record=NbCacheRecord(pk=1), artifacts=NbArtifacts(paths=0))"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_bundle = cache.get_cache_bundle(1)\n",
    "nb_bundle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'cells': [{'cell_type': 'code',\n",
       "   'execution_count': 1,\n",
       "   'metadata': {},\n",
       "   'outputs': [{'name': 'stdout', 'output_type': 'stream', 'text': '1\\n'}],\n",
       "   'source': 'a=1\\nprint(a)'}],\n",
       " 'metadata': {'kernelspec': {'display_name': 'Python 3',\n",
       "   'language': 'python',\n",
       "   'name': 'python3'},\n",
       "  'language_info': {'codemirror_mode': {'name': 'ipython', 'version': 3},\n",
       "   'file_extension': '.py',\n",
       "   'mimetype': 'text/x-python',\n",
       "   'name': 'python',\n",
       "   'nbconvert_exporter': 'python',\n",
       "   'pygments_lexer': 'ipython3',\n",
       "   'version': '3.6.1'},\n",
       "  'test_name': 'notebook1'},\n",
       " 'nbformat': 4,\n",
       " 'nbformat_minor': 2}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_bundle.nb"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Trying to add a notebook to the cache that matches an existing one will result in a error, since the cache ensures that all notebook hashes are unique:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "CachingError",
     "evalue": "Notebook already exists in cache and overwrite=False.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mCachingError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/t2/xbl15_3n4tsb1vr_ccmmtmbr0000gn/T/ipykernel_99993/3576020660.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m record = cache.cache_notebook_file(\n\u001b[0m\u001b[1;32m      2\u001b[0m     \u001b[0mpath\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mPath\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"example_nbs\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"basic.ipynb\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m )\n",
      "\u001b[0;32m~/Documents/GitHub/jupyter-cache/jupyter_cache/cache/main.py\u001b[0m in \u001b[0;36mcache_notebook_file\u001b[0;34m(self, path, uri, artifacts, data, check_validity, overwrite)\u001b[0m\n\u001b[1;32m    268\u001b[0m         \"\"\"\n\u001b[1;32m    269\u001b[0m         \u001b[0mnotebook\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnbf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnbf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mNO_CONVERT\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 270\u001b[0;31m         return self.cache_notebook_bundle(\n\u001b[0m\u001b[1;32m    271\u001b[0m             CacheBundleIn(\n\u001b[1;32m    272\u001b[0m                 \u001b[0mnotebook\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Documents/GitHub/jupyter-cache/jupyter_cache/cache/main.py\u001b[0m in \u001b[0;36mcache_notebook_bundle\u001b[0;34m(self, bundle, check_validity, overwrite, description)\u001b[0m\n\u001b[1;32m    213\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mpath\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexists\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    214\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0moverwrite\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 215\u001b[0;31m                 raise CachingError(\n\u001b[0m\u001b[1;32m    216\u001b[0m                     \u001b[0;34m\"Notebook already exists in cache and overwrite=False.\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    217\u001b[0m                 )\n",
      "\u001b[0;31mCachingError\u001b[0m: Notebook already exists in cache and overwrite=False."
     ]
    }
   ],
   "source": [
    "record = cache.cache_notebook_file(\n",
    "    path=Path(\"example_nbs\", \"basic.ipynb\")\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we load a notebook external to the cache, then we can try to match it to one\n",
    "stored inside the cache:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'cells': [{'cell_type': 'markdown',\n",
       "   'metadata': {},\n",
       "   'source': '# a title\\n\\nsome text\\n'},\n",
       "  {'cell_type': 'code',\n",
       "   'execution_count': 1,\n",
       "   'metadata': {},\n",
       "   'source': 'a=1\\nprint(a)',\n",
       "   'outputs': [{'name': 'stdout', 'output_type': 'stream', 'text': '1\\n'}]}],\n",
       " 'metadata': {'test_name': 'notebook1',\n",
       "  'kernelspec': {'display_name': 'Python 3',\n",
       "   'language': 'python',\n",
       "   'name': 'python3'},\n",
       "  'language_info': {'codemirror_mode': {'name': 'ipython', 'version': 3},\n",
       "   'file_extension': '.py',\n",
       "   'mimetype': 'text/x-python',\n",
       "   'name': 'python',\n",
       "   'nbconvert_exporter': 'python',\n",
       "   'pygments_lexer': 'ipython3',\n",
       "   'version': '3.6.1'}},\n",
       " 'nbformat': 4,\n",
       " 'nbformat_minor': 2}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "notebook = nbf.read(str(Path(\"example_nbs\", \"basic.ipynb\")), 4)\n",
    "notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbCacheRecord(pk=1)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.match_cache_notebook(notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebooks are matched by a hash based only on aspects of the notebook that will affect its execution (and hence outputs). So changing text cells will match the cached notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "notebook.cells[0].source = \"change some text\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbCacheRecord(pk=1)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.match_cache_notebook(notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But changing code cells will result in a different hash, and so will not be matched:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "notebook.cells[1].source = \"change some source code\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "tags": [
     "raises-exception"
    ]
   },
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'Cache record not found for NB with hashkey: 07e6a47c8c180cb7851ede6dbb088769'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/t2/xbl15_3n4tsb1vr_ccmmtmbr0000gn/T/ipykernel_99993/941642554.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcache\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmatch_cache_notebook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnotebook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/Documents/GitHub/jupyter-cache/jupyter_cache/cache/main.py\u001b[0m in \u001b[0;36mmatch_cache_notebook\u001b[0;34m(self, nb)\u001b[0m\n\u001b[1;32m    333\u001b[0m         \"\"\"\n\u001b[1;32m    334\u001b[0m         \u001b[0m_\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhashkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_hashed_notebook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 335\u001b[0;31m         \u001b[0mcache_record\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mNbCacheRecord\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecord_from_hashkey\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhashkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    336\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mcache_record\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    337\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/Documents/GitHub/jupyter-cache/jupyter_cache/cache/db.py\u001b[0m in \u001b[0;36mrecord_from_hashkey\u001b[0;34m(hashkey, db)\u001b[0m\n\u001b[1;32m    158\u001b[0m             )\n\u001b[1;32m    159\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mresult\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 160\u001b[0;31m                 raise KeyError(\n\u001b[0m\u001b[1;32m    161\u001b[0m                     \u001b[0;34m\"Cache record not found for NB with hashkey: {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mhashkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    162\u001b[0m                 )\n",
      "\u001b[0;31mKeyError\u001b[0m: 'Cache record not found for NB with hashkey: 07e6a47c8c180cb7851ede6dbb088769'"
     ]
    }
   ],
   "source": [
    "cache.match_cache_notebook(notebook)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To understand the difference between an external notebook, and one stored in the cache, we can 'diff' them:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nbdiff\n",
      "--- cached pk=1\n",
      "+++ other: \n",
      "\u001b[34m## inserted before nb/cells/0:\u001b[0m\n",
      "\u001b[32m+  code cell:\n",
      "\u001b[32m+    execution_count: 1\n",
      "\u001b[32m+    source:\n",
      "\u001b[32m+      change some source code\n",
      "\u001b[32m+    outputs:\n",
      "\u001b[32m+      output 0:\n",
      "\u001b[32m+        output_type: stream\n",
      "\u001b[32m+        name: stdout\n",
      "\u001b[32m+        text:\n",
      "\u001b[32m+          1\n",
      "\n",
      "\u001b[0m\u001b[34m## deleted nb/cells/0:\u001b[0m\n",
      "\u001b[31m-  code cell:\n",
      "\u001b[31m-    execution_count: 1\n",
      "\u001b[31m-    source:\n",
      "\u001b[31m-      a=1\n",
      "\u001b[31m-      print(a)\n",
      "\u001b[31m-    outputs:\n",
      "\u001b[31m-      output 0:\n",
      "\u001b[31m-        output_type: stream\n",
      "\u001b[31m-        name: stdout\n",
      "\u001b[31m-        text:\n",
      "\u001b[31m-          1\n",
      "\n",
      "\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "print(cache.diff_nbnode_with_cache(1, notebook, as_str=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we cache this altered notebook, note that this will not remove the previously cached notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbCacheRecord(pk=2)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nb_bundle = CacheBundleIn(\n",
    "    nb=notebook,\n",
    "    uri=Path(\"example_nbs\", \"basic.ipynb\"),\n",
    "    data={\"tag\": \"mytag\"}\n",
    ")\n",
    "cache.cache_notebook_bundle(nb_bundle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ID  Origin URI    Created           Accessed          Hashkey\n",
      "----  ------------  ----------------  ----------------  --------------------------------\n",
      "   2  basic.ipynb   2022-01-12 15:16  2022-01-12 15:16  07e6a47c8c180cb7851ede6dbb088769\n",
      "   1  basic.ipynb   2022-01-12 15:15  2022-01-12 15:16  94c17138f782c75df59e989fffa64e3a\n"
     ]
    }
   ],
   "source": [
    "print(tabulate_cache_records(\n",
    "    cache.list_cache_records(), path_length=1, hashkeys=True\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebooks are retained in the cache, until the cache limit is reached,\n",
    "at which point the oldest notebooks are removed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1000"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.get_cache_limit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache.change_cache_limit(100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(use/api/project)=\n",
    "\n",
    "## Staging Notebooks for Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebooks can be staged, by adding the path as a stage record.\n",
    "\n",
    "```{important}\n",
    "This does not physically add the notebook to the cache,\n",
    "merely store its URI, for later use.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbProjectRecord(pk=1)"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "record = cache.add_nb_to_project(Path(\"example_nbs\", \"basic.ipynb\"))\n",
    "record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'uri': '/Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic.ipynb',\n",
       " 'assets': [],\n",
       " 'created': datetime.datetime(2022, 1, 12, 15, 16, 27, 64960),\n",
       " 'traceback': '',\n",
       " 'read_data': {'name': 'nbformat', 'type': 'plugin'},\n",
       " 'pk': 1}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "record.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the staged notbook relates to one in the cache, we will be able to retrieve the cache record:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbCacheRecord(pk=1)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.get_cached_project_nb(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ID  URI                      Reader    Added             Status\n",
      "----  -----------------------  --------  ----------------  --------\n",
      "   1  example_nbs/basic.ipynb  nbformat  2022-01-12 15:16  ✅ [1]\n"
     ]
    }
   ],
   "source": [
    "print(tabulate_project_records(\n",
    "    cache.list_project_records(), path_length=2, cache=cache\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also retrieve a *merged* notebook.\n",
    "This is a copy of the source notebook with the following added to it from the cached notebook:\n",
    "\n",
    "- Selected notebook metadata keys (generally only those keys that affect its execution)\n",
    "- All code cells, with their outputs and metadata \n",
    "  (only selected metadata can be merged if `cell_meta` is not `None`)\n",
    "  \n",
    "In this way we create a notebook that is *fully* up-to-date for both its code and textual content:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1,\n",
       " {'cells': [{'cell_type': 'markdown',\n",
       "    'metadata': {},\n",
       "    'source': '# a title\\n\\nsome text\\n'},\n",
       "   {'cell_type': 'code',\n",
       "    'execution_count': 1,\n",
       "    'metadata': {},\n",
       "    'outputs': [{'name': 'stdout', 'output_type': 'stream', 'text': '1\\n'}],\n",
       "    'source': 'a=1\\nprint(a)'}],\n",
       "  'metadata': {'test_name': 'notebook1',\n",
       "   'kernelspec': {'display_name': 'Python 3',\n",
       "    'language': 'python',\n",
       "    'name': 'python3'},\n",
       "   'language_info': {'codemirror_mode': {'name': 'ipython', 'version': 3},\n",
       "    'file_extension': '.py',\n",
       "    'mimetype': 'text/x-python',\n",
       "    'name': 'python',\n",
       "    'nbconvert_exporter': 'python',\n",
       "    'pygments_lexer': 'ipython3',\n",
       "    'version': '3.6.1'}},\n",
       "  'nbformat': 4,\n",
       "  'nbformat_minor': 2})"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.merge_match_into_file(\n",
    "    cache.get_project_record(1).uri,\n",
    "    nb_meta=('kernelspec', 'language_info', 'widgets'),\n",
    "    cell_meta=None\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we add a notebook that cannot be found in the cache, it will be listed for execution:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbProjectRecord(pk=2)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "record = cache.add_nb_to_project(Path(\"example_nbs\", \"basic_failing.ipynb\"))\n",
    "record"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache.get_cached_project_nb(2)  # returns None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[NbProjectRecord(pk=2)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.list_unexecuted()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ID  URI                              Reader    Added             Status\n",
      "----  -------------------------------  --------  ----------------  --------\n",
      "   1  example_nbs/basic.ipynb          nbformat  2022-01-12 15:16  ✅ [1]\n",
      "   2  example_nbs/basic_failing.ipynb  nbformat  2022-01-12 15:17  -\n"
     ]
    }
   ],
   "source": [
    "print(tabulate_project_records(\n",
    "    cache.list_project_records(), path_length=2, cache=cache\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To remove a notebook from the staging area:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "cache.remove_nb_from_project(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ID  URI                              Reader    Added             Status\n",
      "----  -------------------------------  --------  ----------------  --------\n",
      "   2  example_nbs/basic_failing.ipynb  nbformat  2022-01-12 15:17  -\n"
     ]
    }
   ],
   "source": [
    "print(tabulate_project_records(\n",
    "    cache.list_project_records(), path_length=2, cache=cache\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(use/api/execute)=\n",
    "\n",
    "## Execution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we have some staged notebooks:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "NbProjectRecord(pk=2)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.clear_cache()\n",
    "cache.add_nb_to_project(Path(\"example_nbs\", \"basic.ipynb\"))\n",
    "cache.add_nb_to_project(Path(\"example_nbs\", \"basic_failing.ipynb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ID  URI                              Reader    Added             Status\n",
      "----  -------------------------------  --------  ----------------  --------\n",
      "   1  example_nbs/basic.ipynb          nbformat  2022-01-12 15:17  -\n",
      "   2  example_nbs/basic_failing.ipynb  nbformat  2022-01-12 15:17  -\n"
     ]
    }
   ],
   "source": [
    "print(tabulate_project_records(\n",
    "    cache.list_project_records(), path_length=2, cache=cache\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can select an executor (specified as entry points) to execute the notebook.\n",
    "\n",
    "```{tip}\n",
    "To view the executors log, make sure logging is enabled,\n",
    "or you can parse a logger directly to `load_executor()`.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'local-parallel', 'local-serial', 'temp-parallel', 'temp-serial'}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list_executors()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "JupyterExecutorLocalSerial(cache=JupyterCacheBase('/Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/.jupyter_cache'))"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from logging import basicConfig, INFO\n",
    "basicConfig(level=INFO)\n",
    "\n",
    "executor = load_executor(\"local-serial\", cache=cache)\n",
    "executor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling `run_and_cache()` will run all staged notebooks that do not already have matches in the cache.\n",
    "It will return a dictionary with lists for:\n",
    "\n",
    "\n",
    "- **succeeded**: The notebook was executed successfully with no (or only expected) exceptions\n",
    "- **excepted**: A notebook cell was encountered that raised an unexpected exception\n",
    "- **errored**: An exception occured before/after the actual notebook execution\n",
    "\n",
    "```{tip}\n",
    "Code cells can be tagged with `raises-exception` to let the executor known that\n",
    "a cell *may* raise an exception (see [this issue on its behaviour](https://github.com/jupyter/nbconvert/issues/730)).\n",
    "```\n",
    "\n",
    "```{note}\n",
    "You can use the `filter_uris` and/or `filter_pks` options to only run selected staged notebooks.\n",
    "You can also specify the timeout for execution in seconds using the `timeout` option.\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:jupyter_cache.executors.base:Executing 2 notebook(s) in serial\n",
      "INFO:jupyter_cache.executors.base:Executing: /Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic.ipynb\n",
      "INFO:jupyter_cache.executors.base:Execution Successful: /Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic.ipynb\n",
      "INFO:jupyter_cache.executors.base:Executing: /Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic_failing.ipynb\n",
      "WARNING:jupyter_cache.executors.base:Execution Excepted: /Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic_failing.ipynb\n",
      "CellExecutionError: An error occurred while executing the following cell:\n",
      "------------------\n",
      "raise Exception('oopsie!')\n",
      "------------------\n",
      "\n",
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)\n",
      "\u001b[0;32m/var/folders/t2/xbl15_3n4tsb1vr_ccmmtmbr0000gn/T/ipykernel_1308/340246212.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n",
      "\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'oopsie!'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0m\n",
      "\u001b[0;31mException\u001b[0m: oopsie!\n",
      "Exception: oopsie!\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "ExecutorRunResult(succeeded=['/Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic.ipynb'], excepted=['/Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic_failing.ipynb'], errored=[])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result = executor.run_and_cache()\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Successfully executed notebooks will be added to the cache, and data about their execution (such as time taken) will be stored in the cache record:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[NbCacheRecord(pk=1)]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cache.list_cache_records()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'description': '',\n",
       " 'hashkey': '94c17138f782c75df59e989fffa64e3a',\n",
       " 'created': datetime.datetime(2022, 1, 12, 15, 17, 45, 471862),\n",
       " 'accessed': datetime.datetime(2022, 1, 12, 15, 17, 45, 471871),\n",
       " 'data': {'execution_seconds': 1.8344826350000005},\n",
       " 'uri': '/Users/chrisjsewell/Documents/GitHub/jupyter-cache/docs/using/example_nbs/basic.ipynb',\n",
       " 'pk': 1}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "record = cache.get_cache_record(1)\n",
    "record.to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notebooks which failed to run will **not** be added to the cache,\n",
    "but details about their execution (including the exception traceback)\n",
    "will be added to the stage record:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/jupyter_cache/executors/utils.py\", line 58, in single_nb_execution\n",
      "    executenb(\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/site-packages/nbclient/client.py\", line 1093, in execute\n",
      "    return NotebookClient(nb=nb, resources=resources, km=km, **kwargs).execute()\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/site-packages/nbclient/util.py\", line 84, in wrapped\n",
      "    return just_run(coro(*args, **kwargs))\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/site-packages/nbclient/util.py\", line 62, in just_run\n",
      "    return loop.run_until_complete(coro)\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/site-packages/nest_asyncio.py\", line 81, in run_until_complete\n",
      "    return f.result()\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/asyncio/futures.py\", line 178, in result\n",
      "    raise self._exception\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/asyncio/tasks.py\", line 280, in __step\n",
      "    result = coro.send(None)\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/site-packages/nbclient/client.py\", line 559, in async_execute\n",
      "    await self.async_execute_cell(\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/site-packages/nbclient/client.py\", line 854, in async_execute_cell\n",
      "    self._check_raise_for_error(cell, exec_reply)\n",
      "  File \"/Users/chrisjsewell/Documents/GitHub/jupyter-cache/.tox/py38/lib/python3.8/site-packages/nbclient/client.py\", line 756, in _check_raise_for_error\n",
      "    raise CellExecutionError.from_cell_and_msg(cell, exec_reply_content)\n",
      "nbclient.exceptions.CellExecutionError: An error occurred while executing the following cell:\n",
      "------------------\n",
      "raise Exception('oopsie!')\n",
      "------------------\n",
      "\n",
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m\n",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)\n",
      "\u001b[0;32m/var/folders/t2/xbl15_3n4tsb1vr_ccmmtmbr0000gn/T/ipykernel_1308/340246212.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n",
      "\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'oopsie!'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0m\n",
      "\u001b[0;31mException\u001b[0m: oopsie!\n",
      "Exception: oopsie!\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "record = cache.get_project_record(2)\n",
    "print(record.traceback)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have two staged records, and one cache record:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ID  URI                              Reader    Added             Status\n",
      "----  -------------------------------  --------  ----------------  --------\n",
      "   1  example_nbs/basic.ipynb          nbformat  2022-01-12 15:17  ✅ [1]\n",
      "   2  example_nbs/basic_failing.ipynb  nbformat  2022-01-12 15:17  ❌\n"
     ]
    }
   ],
   "source": [
    "print(tabulate_project_records(\n",
    "    cache.list_project_records(), path_length=2, cache=cache\n",
    "))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  ID  Origin URI    Created           Accessed          Hashkey\n",
      "----  ------------  ----------------  ----------------  --------------------------------\n",
      "   1  basic.ipynb   2022-01-12 15:17  2022-01-12 15:17  94c17138f782c75df59e989fffa64e3a\n"
     ]
    }
   ],
   "source": [
    "print(tabulate_cache_records(\n",
    "    cache.list_cache_records(), path_length=1, hashkeys=True\n",
    "))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Timeout\n",
    "A **timeout** argument can also be passed to `run_and_cache()` which takes value in seconds.\n",
    "Alternatively, timeout can also be specified inside the notebook metadata:\n",
    "\n",
    "```\n",
    "'execution': {\n",
    "   'timeout': 30\n",
    " }\n",
    "```\n",
    "```{note}\n",
    "Timeout specified in notebook metadata will take precedence over the one passed as an argument to `run_and_cache()`.\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "interpreter": {
   "hash": "8398b65b1e6feb38b0506d5ab1aedf8bf63748a9844a9c81ed9242850234e24f"
  },
  "kernelspec": {
   "display_name": "Coconut",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
